Since the results of existing intelligent algorithms of pulmonary nodule detection only predict the positions of nodules and cannot give semantical interpretations which are well known to doctors in clinical diagnosis, such as “lobulation”, “texture” and “spiculation”, a pulmonary nodule detection method with semantic feature score was proposed. Eight semantic features—subtlety, internal structure, lobulation, spiculation, margin, calcification, sphericity and texture were embedded into the Region Proposal Network (RPN) of Faster R-CNN, a new anchor box mechanism was designed, a fully connected network was added to realize the regression learning of semantic features, and the semantic scores were used as auxiliary information to realize the joint learning of pulmonary nodule detection and semantic prediction by training with Faster R-CNN. The proposed method was evaluated on the LIDC/IDRI dataset. Results show that the accuracy of pulmonary nodule localization is 91.2%, and the accuracy, sensitivity and specificity of benign and malignant classification are 81%, 91.2% and 70.8% respectively. On 8 semantic feature scores, the difference between doctors is 0.58±0.78 (mean absolute error±standard deviation), the proposed method achieves the difference of 0.62±1.03 with doctors, which is familiar to the former one. These results demonstrate that the modified network has good prediction accuracy and semantic feature prediction, and facilitates the understanding and clinical interpretations of machine prediction results for doctors.